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I. Executive Summary 


The Conclusions and Recommendations resulting from this study are 


summarized in the following two sections. 


A. General 


1. 


2. 


3. 


The Intelligence Community must Task - as required by Executive Order. 


Tasking without assessment is not workable. There would be no rational 
grounds upon which management could base the necessary decisions regard- 
ing the direction of resources to respond to Tasking received or to 
undertake later retasking in view of results obtained or not obtained. 


A HUMINT Assessment System (HAS) is necessary in order to support a 
fully operational HUMINT Tasking System. There is, at present, no 
assessment system that could support the level of tasking and retasking 
expected within the next year. 


However, the experimentation conducted during this special study has 
established that the basics for a HUMINT Assessment System do exist 
or could be quickly set in place. These basics are detailed in the 
following section. 


B. Specific 


l. 


The Intelligence Community has in-hand the following with respect to a 
system for assessment of HUMINT reporting in response to tasking (Tasking 
Plans): 

(a) A workable approach to assessment based upon the concept of providing 
semi-centralized linkages and/or supplementation to existing 
assessment activities and, where such activities do not exist, 
providing the entire assessment effort necessary. This seni- 
centralized approach makes use of.existing assessment activities, 
minimizes duplication of efforts and impacts on customer/consumer 
time, coordinates and supplements on-going assessment activities, 
and provides the necessary level of assessment where none currently 
exists. 


(b) A tested approach to assessment resulting from the experience 
obtained by actual, successful use of the semi-centralized 
approach in experimentation conducted over the past six (6) months 
related to this study. 


(c) An acceptable assessment methodology - the customer/consumer 
interview model. The customer interview model has been tested and 
refined through several years of use within the DDO. It is an 
acceptable evaluation/assessment methodology which can be still 
improved (see Recommendations). The metholology has been further 
refined and successfully utilized during this special study on an 
experimental basis. 
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2. The Intelligence Community needs to: 
- (a) Establish a system for the assessment of HUMINT reporting responding 
to Tasking. In establishing this HUMINT Assessment System the 
Community should build upon the tested approach (semi-centralized, 
linkage) and methodology (customer interview) which are in-hand. 


(b) Refine and further define the HUMINT Assessment Systems concepts 
by completing a formal Evaluability Assessment of the system 
(Appendix C). In this regard it should be noted that the concepts 
upon which the HUMINT Assessment System should be developed have 
already been partially explored during this study: , 

(1) existing HUMINT assessment activities representing potential 
entities for linkage into the assessment system have been 
identified and their level of evaluaton activity determined 
(Figure V-1). 

(ii) the system structure and flow of National Tasking and Response/ 
HUMINT was outlined and measurement points, assessment 
elements, and assessment actions were determined (Appendix D). 

(iii) the objectives of HUMINT Response Assessment were refined and 
set down in writing (Appendix E). 


(c) Provide further evaluation/assessment support for: 

(i) management decision making related to trade-offs, resource 
reallocaton, "doing more with less”. 

(ii) budgetary preparation and defense with quantification of 
impacts and results related to resource levels. 

(d) Continue refinement of the Customers Interview Method with specific 
attention to establishing statistically supportable claims related 
to the survey results, (through random sampling techniques). 

The Intelligence Community has a great deal of work ahead in developing the 
role and contribution that assessment/evaluation can play in enhancing management 
decision making and supporting Tasking. However, the basic approaches and 
methodologies do exist and have been tested. The next step is to bring these 
basic elements together into an assessment system. A good portion of the pre- 
liminary work to establish such a system has been conducted during the course of 


this experimental study. The remaining and recommended activities are seen as 


plausible, workable, and positive contributions. 
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II. Objective of Study - Introduction and Background 

The objectives of the study undertaken by Madden, Madden, and Associates, 
Inc. (MMA) in conjunction with staff of the HUMINT Tasking Office - Assessment 
Branch (HTO-AB) were: (a) to research evaluation efforts or techniques related 
to or applicable to intelligence reporting evaluation; (b) to prepare a study 
that would document and describe such efforts and; (c) if previously or currently 
utilized, explore the reasons for their respective successes or failures. 
The research effort would include non-Intelligence Community (IC) 
.departments, sveieieds aid /oe-uites within its scope in an attempt to 

‘ discover and consider any evaluation efforts adaptable to IC use. Recom- 

mendations regarding on~going and possible IC evaluation techniques, tools, 
methods, and approaches would be made. The evaluation method(s) for a 
specific, IC selected area of concern (HUMINT related) would be used as a 
“demonstration” of the recommendations. 
A. Research of evaluation efforts and techniques: 

The study would identify and review current evaluation efforts 
related to HUMINT in the Intelligence Community (IC). Included in the study 
would be State, Commerce, and other appropriate agencies or departments outside 
the Intelligence Community. The scope of this “search” would include Federal, 
non-Federal, governmental, non-governmental, and/or other evaluation efforts 
that may be applicable to IC evaluation. The search process would include 
on-site visits and discussions as appropriate. 

(1) A copy of existing documentation or guides for each evaluation 

effort, if available, would be obtained and filed ina central 


repository for access by IC evaluation (or interested) staff 
as appropriate. 
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(2) Sample copies of evaluation reports and/or other outputs from 
evaluation efforts would also be filed. 
Note - An appropriate organizational location for such a 


central file of information on evaluation would be 
selected. 


(3) The flow of current IC evaluation reporting [source to user(s) 
of such reporting] would be identified and detailed in appro- 
priate charts. 


(4) Previous IC evaluation efforts would be searched for and, if 
found, identified as to type of evaluation and pertinent 
related data (who, where, when, extent of effort, use, users, 
etc.). Existing documentation and samples, if available, 
would be obtained and centrally stored. 


The Contractor (MMA) would provide during and at the conclusion of 
the special study project the following tangible products in written 
and/or briefing or seminar format as indicated: 


(1) Recommendation as to central file and repository for data 
and documentation concerning evaluation (written, briefing 
as warranted). 


(2) Listing and identifying current, on-going evaluation efforts 
(IC and relevant non-IC) - who, where, when, what, extent, 
contact person, users, etc. The list would refer to 
available documentation (description, examples, etc.) in 
the central repository if such was on file (written). 


(3) Listing of previous IC evaluation efforts identified during 
this research. 


(4) Description (narrative) and chart(s) of source through user(s) 
of IC evaluation effort reporting. 


B. Evaluation of HUMINT evaluation efforts and techniques: 


The study would review and evaluate the previous and on-going 
IC evaluation efforts: 


(1) This review would utilize the framework and concepts regarding 
evaluation presented in the:: 
- Seminar on Evaluating Human Source Intelligence, 
11-12 December 1978, Center for the Study of 
Intelligence, CIA. 
- Seminar on Tools and Techniques for Evaluating 
Human Source Intelligence, 31 May 1979, Center 
for the Study of Intelligence, CIA. 
The conceptual framework for this review is presented in Part 
III- (Conceptual Framework for Evaluation) of this report. 
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(2) The review would be a positive, constructive critique of the 
previous and on-going evaluation efforts. It would provide: 


(a) a description of the relationship to the framework 
for evaluation developed in the above seminars 
including comments as to variations from the frame- 
work and evaluation as to the implications (+ or -) 
of such variation. 


(b) an analysis of the reasons for the respective success 
and/or failure of the efforts; including “lessons to 
be learned" and concepts that could be fruitfully 
adapted from others. This would include IC and non-IC 
evaluation efforts, techniques, and tools. 


(c) specific recommendations for on-going efforts and 

suggested organizational adoptions of “other's” strengths. 
The “other” organizational evaluation efforts considered 
for adoption would be as wide ranging as possible ~- not 
necessarily limited to IC, Federal, or governmental 
evaluation approaches. 


(d) recommendations for evaluation would be made for a specific 


HUMINT reporting area (chosen by the IC) as a "demonstration" 
of the recommended evaluation efforts. 
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Contractor Qualifications 


A. General 


Mr. Lynn P. Madden is Senior Consultant and President of Madden, 
Madden, and Associates, Inc. a management consulting and training firm. 
Madden, Madden and Associates, Inc. (MMA) has an extensive background in 
providing training, evaluation, consultation, and organizational and 
management development services to the public sector. Since its founding 
in 1971, MMA has: 

1. met all objectives of projects undertaken. 

2. met all project schedules and deadlines. 

3. never exceeded budgeted contract costs. 
This is due to the continuous internal application of the management 
concepts and techniques MMA makes available in services to clients. 
Since 1971, Madden, Madden and Associates, Inc. has conducted 
between 85 and 125 seminars per year for public sector organizations. 
The seminars range from two (2) days to two (2) weeks in duration. 
MMA specializes in consultation and training of upper and middle manage- 
ment in government organizations, especially at the state and Federal 
level. MMA conducts seminars and provides consulting services for the 
states along the east coast, Federal Governmental Agencies, ania the 


United Nations both in New York and Geneva, Switzerland. 


B. Related Experience 


this 


Assignments undertaken by Mr. Madden that are of direct relevance to 
project include: 
1. Activity (Effectiveness) Assessment in Government —- 
a 2 day program for middle and upper level management 
in the concepts and application of evaluation to 


governmental activities. The program was developed 
by Mr. Madden and has been conducted for: 
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(a) U.S. Civil Service Commission (OPM) - 
Washington, DC., Boston, St. Louis since 
1974. Approximately 10-15 seminars per year 
have been conducted —- 6 are currently 
scheduled for 1980. 

(b) United Nations - Geneva, Switzerland since 
1977, once per year for upper level UN 
family (UNDP, ILU, ITO, UN-Secretariat, etc.) 
management. 

(c) New York State - Public Service Commission, 
Department of Health, Department of Social 
Services. 

(d) State of Maryland - Department of Human 
Resources, Management Development Center. 

(e) Pennsylvania Department of Public Welfare - 
Boards of Local Assistance. 


2. Project Management - 2 to 5 day seminars for pro ject/ 
program managers developed by MMA and conducted for: 

(a) US Department of Transportation, Transportation 
System Research Center, Cambridge, Massachusetts. 

-(b) US Naval Weapons Laboratory, Dalhgren, Virginia. 

(c) US Airforce - MAC, Dover AFB. 

(d) Department of Transportation State of Maryland. 

(e) US Civil Service Commission (OPM) - Philadelphia, 
Boston. 


3. Operations Research Techniques - 2 to 4 day seminars 
developed and conducted for: 
(a) OPM - Philadelphia, Boston. 
(b) New York State Civil Service Department 
(c) United Nations - Geneva, Switzerland. 


4. Evaluation of Human Source Intelligence - a 2 day seminar 
conducted at the Center for the Study of Intelligence 
(11-12) December 1978). The seminar explored concepts of 
evaluation applicable to HUMINT reporting and the 
current status of reporting in the Intelligence Com- 
munity. Participant reactions and recommendations 
were set forth in a report on the seminar prepared by 
the Center. 


_5. Tools and Techniques for Evaluating Human Source 
Intelligence - a 1 day workshop conducted at the Center 
for the Study of Intelligence (31 May 1979). The 
_ workshop explored concepts, methods, and tools of 
evaluation and their application to HUMINT reporting. 
As a direct result of the above consulting and training seminars Mr. 

Madden comes in contact with hundreds of management and professional staff 
(from different agencies, organizations, and levels of government, private 


sector firms, and academic) who have a concern with evaluation. Their concern 
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ranges from conceptual, theoretical concepts to practical, day-to-day 


applications as operating managers. In addition to this on-going contact 
and interaction, Mr. Madden is actively involved in reading the literature 
and maintaining a current professional working knowledge in the field of 
evaluation, assessment of efficiency, and workload standards. 
C. Education and Experience 
Mr. Madden's formal education and previous experience are also relevant 
to consultation on evaluation of intelligence reporting: 
Education - Bachelor of Electrical Engineering - 
Syracuse University 
Master of Science, Industrial Administration — 
Union College, Schenectady, New York 
Master of Business Administration, Public Systems ~ 
Union College 
Related 
Experience - Engineer - International Business Machines with 
project management responsibilities. 
Analyst - US Army Reserves, 454 Military 
Intelligence Detachment (6 years) - 


working at the Pentagon, Arlington Hall, 
and CIA Headquarters. 
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IV. Conceptual Framework For Evaluation 
A. Concepts and Terms 


Analysis = future oriented - which alternative/option to choose 
and why? 


Evaluation = present to past orientation - what have we got? How 
have we done? 


' Evaluation is concerned with three related, but distinct aspects that 
“build” upward in a hierarchal fashion: 


Effectiveness 


How these aspects relate can be shown in a general “model” of any activity 


(field reporting for example) as a flow processing system that “produces” 


certain goods or services which are then used by end "users/consumers”. 

consumer use results in certain impacts and/or end effects. 

Inputs Outputs 

($,people, Process 

resources) (goods/services) 

Transformed (T) 
into: 
Results/Effects 
(Impacts of various 
types —- i.e., 
Effects lower costs, avoided 
expenses, etc.) 
-9- 
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B. Quality 


Evaluation of Quality would concern itself with the Output(s): 


: - definition of output - what is the unit “produced” (goods, 
service, report)? How will I know an output unit? How will I 
know a unit of reporting? 


- attributes of the output (implicit in the definition of the 
unit is the concept of “quality” - what perceivable attributes 


will be taken as making up an acceptable unit of output)? What 
attributes define an item of reporting? 


The attributes that represent “quality” of the output unit should be, but often 
are not, as specific, measurable, and quantifiable as possible. 


The evaluation of quality requires that: 


1. The output units (goods and/or services, reporting) be identified 
and defined: “The manager of the decision unit should 
include in a statement of objectives information relating 
to both the services or products to be provided by the 
decision unit and the contribution of that output to 
achievement of the principal missions toward which the 
unit's efforts are directed.” [OMB A-115, 5/5/78] 


2. Quality measures (attributes) be stated and defined —- 
this is implicit in the definition of the output unit. 
The attributes should be as quantifiable and objective 
as possible, however, subjective attributes will often be 
necessary or unavoidable. Note that the quality measures 
(attributes) have similar characteristics to objectives 
and, in fact, may be stated as objectives regarding the 

et unit of output. 


Example - It is the agency objective(s) that all field reporting be: 


- timely 

~ reliable information 

- valuable 

- of useful nature [Ref. DD Form 1480] 


Note - To make the above attributes operational as measures of 
quality they would require: 


- further definition or 
- selection of (reference to) a “judge”. 


The "measurement" of the attribute existence can be either: 
Objective - by some “instrument” independent of 
~~ human judgment. 

Subjective - the measurement involves human 
judgment, i.e., the “instrument” 
is a person (judge) or persons (panel). 
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Taken together the set of attributes defines the output unit. Any thing 
(good/service, report) which meets the minimal level of attributes is by defini- 
tion a unit of output - if it does not, then it is not an output, it is not a report. 

The set of attributes used to evaluate quality consists, in most cases, 
of separate attributes of varying importance to the evaluator or decision 
maker. There are tools available to determine the relative importance of the 
attributes to the evaluator. 
or Efficiency 

Evaluation of Efficiency concerns itself with the relationship between the 
Output(s) and Input(s). There are three general categories of efficiency evalua- 
tion or assessment: 


1. Work Measurement (WM) - relates the human resources (hours, 
months, years) “consumed” in the production of a unit of output. 


2. Unit Cost (UC) - relates the cost of all accountable and allocat- 
able resources to the production of a unit of output. 


3. Productivity Index (PI) - relates any “meaningful” measure of out- 
put to any measure of input. 


Although these three forms of efficiency evaluation may appear in different ways 
there are only the three basic forms given above. 
D. Effectiveness 
Evaluation of Effectiveness requires that the “results” and/or “effects” of 
the outputs be considered. There are, perhaps surprisingly, only three approaches - 
that are available to evaluate effectiveness: 
(a) Numeric ratio - the ratio of measurable results obtained over 
stated objectives. 
(b) Cost-effectiveness - the ratio of measurable results (in some 
unit) over the costs of the inputs. 


(c) Benefit~Cost - the ratio of the benefits ("value") of the re- 
sults over the cost of the inputs. 
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‘In cost-effectiveness analysis the stated objectives should be result-related - 
not output-orientated (often this is not true). Benefit-cost analysis requires 
the “valuation” (via shadow prices) of both resources (inputs) and the results/ 
effects “produced” by the outputs. There are two major issues in the evaluation 
of effectiveness: 

- do your outputs “produce” a result/effect? 


- does your effect have “value”? 


Effectiveness Ratio Yes No 
Cost-Effective Yes y No 
Benefit—-Cost Yes Yes 


Note - In Unit Cost, Cost-Effectiveness, and Benefit-Cost ratios the cost 
portion does not have to be the all-inclusive cost of all of the 
inputs. In fact, in most such ratios the costs used are knowingly 
limited to partial costs (or accepted as not all-inclusive, due to 
unknown, unallocatable, unmeasurable costs recognized as existing). 


E. Evaluation - A Management System 


The three aspects of evaluation provide management with different items of 
information at different organizational levels. Different management levels 
within the intelligence community would require different evaluation informa- 
tion. However, the provision of such information should be approached through 
the development of a hierarchically structured, integrated system where each 
level of evaluation utilizes the evaluation systems, reports, and basic data of 
the previous level. 


Executive, policy,upper 
management. 
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VY. Research of Evaluation Efforts and Techniques 

The study objectives required that evaluation efforts, methods, techniques 
and tools relevant to HUMINT be explored - both within the IC and in other 
agencies, departments, governmental, and non-governmental organizations. On- 
site visits for briefings and interviews were conducted at sixteen (16) Ic 
related entities and five (5) non-IC agencies or departments. On-site visits 
were supplemented by numerous telephone calls and background readings of related 
reports, articles, memos, and books. 

A. Intelligence Community (IC) 

A questionnaire was prepared and distributed as a further supplement to 
the sixteen (16) on-site interviews conducted during this study (see Appendix A). 
Written responses to the questionnaires were received from six (6) activities. 

A total of nineteen (19) separate entities related to HUMINT evaluation within 
the IC were identified as currently existing (18) or previously existing (1). 
The 19 entities and related descriptive data are shown in Figure V-l. The key 
to the descriptive data is included in the Figure (refer to section IV of this 
report for background and definition of terms). HTO(AB) staff have available 
specific contact person information (name, telephone, witeas) for the 18 active 
groups listed. 

Be HUMINT Assessment — HTO(AB) 

The evaluation or assessment of HUMINT conducted by the HTO during this 
study was done through an interface, semi-centralized approach as shown in 
Figure V-2. The HTO(AB) utilized, to the extent possible, existing assessment 
efforts. HTO(AB) provided assessment through ad-hoc, supplementary means in 
those cases where there were no existing efforts. The evaluation was of the 


quality of the reporting as “measured” subjectively by the user/consumer of 
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SYMBOL KEY (Reference Figure V-1) 


= Activities: I. Level of Evaluation: 
Figure V-1 : Sutvey_ of HUMINT Assessment Activities: t = Subjectively measured against stated, defined 


LEVEL OF EVALUATION EFFORT(S) attributes. The measures can be (depending on 

ACTIVITY QUALITY EFFICIENCY EFFECTIVENESS MARKET NATURE SOURCES DATA BASE ENTITY# ’ the tnotance) obtained from: 
: (1) the users/consumeres - vin interviews (69) 

CIA-DDO: e or questionnaires (Q): 


(2) a panel of experts. 


a ee : TH PM: N388:P 0-2;3;DIVS;I  PMF(M;A) 1 (3) a single expert. 
DDO-COMPTR S(1-134) WH; UC; PI CTL i 3 BS 388; 333 ‘ , : (4) the evaluations done by others 
DDO-OPEG 8(1-133;4) wt - TH; PM; BS R;SS;P;0TO O-3;DIVS;1 FILES(M;A) 2 erica = objectively measured va, stated, defined attributes i 
oie 7 +ER THs PM:I3 R:P:0TO 0 FILES(H;A);PSS 3 WM = Work measurement ~ output/staff hours i 
DDO-FREG 8(1-1;4) TI; PM; Ip BS 3P5 aree UC # Unit-cost~output/$ ' 
ge 2 TM: PM; OTO;N;P O:1 FILES(M; A); PSS 4 Pl Productivity index - output weasure/input meaeure ‘ 
NFAC/RES 8(1-15Q) _ om ee a - es EFFECTIVENESS: : 
a f « ER«= Effectiveness ratio - reeult/objective ! 
CIA-COMPTR: c CE = Cost effectiveness ratio - resuite/$ 
4 WM; UC;PI. ER; CTE 8$;PM;1™ P; OTO 0-1;233 - 5 BC = Benefit-cost ratio - benefits/§ 
COMPTR GRP S(&) 3UC; ‘ oR ‘ B CTI = Citation of tangible (seasurable) impacte 
ANALYSTS GR? = (NEW = IN PROCESS OF SET-UP) BS; PH oTO;P 130 Sy 6 


Il. “MARKET for Evaluation Reporting - an indication of the role or t 
DIA: o use the evaluation reports are put to (used for) ~ ie, what type 


of efforts supported by the evaluation: ! 


DIA-DC3 8( 11323334) wt ER; CTL All-see key N;SS;P;0TO 0-8;PROD FILES(M;A) © ? ™ = Top (director, deputy director ‘evel) uanagenent 
® . having agency level (directorate) responsibility. ‘ 
- 3UC;P: ER;CE;CTI =. TM; PM; BS; 1 SS3P;N;0TO 0 HIMS(A) ; PMP 8 F POL = Policy making ~ the set of upper level executives 
RLA-DCA S(1-9) MDG FE Eas SMM pain : involved in debating, resolving, and setting policy. 
u DIA/DM-D 8(1;2;4) wu ER; CTI POL N;SS;0TO O-PROD FILES(M) 9 PM © Program aanagement ~ the set of managers responsible 
rd a for a specific program (implementation, operations, : 
“4 DIA-DP s - ER TM; PM; BS NP I FILES(M) 10 direction/redirection, etc.); includee special studies , 
. + requested by a unit, agency or department. : 
x TM: PM; OTO;P O-12 - lh: : B$ = Budget - the set of managers and staff involved in , 
Ne He & aPMG BS ‘ budgetary preparation and defense, resource allocation, ! 
= - TM;PM-HTO(TB) OTO;N(TP) 130 PILES 12 and fund redirections i 
Hro(as) s(-1) Cr . (rR) . I © Individual consumer - @ single, individual who uses the 
Tao - - ER ™; PM OTO;N O-4;11 - 13- evaluation reporting to carry out his/her misaion; ‘ 
this category includee epecial atudies on a request 
FSIS 8(1-9;1;4) - - PM P;N 130-4 FILES 14 basis for a single person. 
sre} a < FP SET-UP) = —_ - - 15 IIT, NATURE of Evaluation Reportin 
HAVECHIC) Cares Tuznocrs® 0 : Ws" Warrative reports, little or no “statistical” 
« = PM OTO:;N 0 - 16 data included. 
STATE (INR) 5(4) i S$ = Statistical summarization, tabular/avamary data, 
3234 = CTL TM; PM; POL N; OTO O-4;11;1681_ FILES 17 scales, scalar values, charts, etc. 
Focus 8(15254) ¢ ae ; aoe P= Periodic in nature, written/produced at regular 
=| =) - DISCNT~12/78)= PM N;SS I-(PANEL) - 18 acheduled intervale, 
STATESMGT 078/85 PANEL aie) OTO = One~Time-Only written for a specific need, in 
CIA-IG STAF? (4) - - ™ TO O;I-4 - - 9 Tesponse to a particular request for evaluation data. 


IV. SOURCES of Data for Evaluation 
O= Others, outside the evaluation unit ~ fe, 
consumers, usere, other evaluation units, etc. 
I= Internal ~ the data is obtained through studies, 
measures, ctc. conducted inside the evaluation unit. 
Note ~ number indfcated is the “entity #” assigned on chart. 


VY. DATA BASE 
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Figure V-2: Interface Approach 
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reporting being interviewed. Citations of tangible impacts were recorded 


as mentioned. 
C. Non-Community Activities 
On-site visits were undertaken at five (5) non-IC entities. The 
visits were supplemented by telephone calls and numerous readings of 
written materials. Relevant articles, books, and documents were 
obtained for this study's files (HTO-AB) - see Reference List attached. 
D. Federal Policy on Evaluation 
| Relevant Federal policy and guidance related to evaluation was sought. 

Specific inquiry into such policy was made during visits to OMB and GAO. 
The following relevant documents were obtained for the study files (see 
extracts in Appendix B): 

1. .OMB Circular A-11 

2. OMB Circular A-115 


3. OMB Circular A-117 


4. GAO - “Comprehensive Approach for Planning and 
Conducting A Program Results Review (Exposure Draft)" - 
6/78. 


5. GAO - “Guidelines for Economy & Efficiency Audits of Federally 
Assisted Programs (Exposure Draft)" 6/79. 
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VI. Conclusions 


mm) 


A. Intelligence Community (IC) 


” 


1. Current evaluation efforts in IC related to HUMINT have been identified and 
reviewed (19 “entities” and 3 “data bases”): 


(a) The evaluation efforts are distinct activities conducted by 
separated organizational entities having little coordination and/ 
or interconnection of efforts and reporting. 

(b) Most efforts are of the special, “one-time-only-study” nature 
produced for a single decision maker/user. 

(c) Most efforts deal with “quality” of reporting only and little or 
no “efficiency” or "effectiveness" evaluation. 

(d) The survey included individuals responsible for budget preparation 
and defense. A number of them expressed a pressing need for 
evaluation information of a type and amount beyond that now coming 

_. to then. 

(e) A need for increased quantification in evaluation is felt 
through out the IC - both in budget (resource) and program 
management areas. 


2. Previous IC evaluation efforts have been identified. They are mainly 
“one-time-only", special studies conducted for specific needs and focus on 
evaluation of reporting “quality”. 


3. There is no Intelligence Community (IC) coordinated, systematic reporting/ 
response assessment system in existence to support the HUMINT Tasking System 
as it is currently constituted. Assessment to support HUMINT Tasking re- 
quires supplementary efforts (such as conducted in this special study, using 
the Customer Interview Model) building upon such in-place efforts as do exist. 


4. The previous and existing evaluation activities (mainly focussed on evalua- 
tion of "Quality" for specific, “local” management use) are not, at this 
time, adequate for the provision of responses for: 

(a) Community level tasking assessment. 

(b) Resource allocation concerns. 

(c) Budgetary defense and justification. 
To respond to such concerns the evaluation activities will need to “raise” 
the level of efforts to “Efficiency” and "Effectiveness" evaluation (see 
section IV). 


B. HUMINT - HTO 


1. HUMINT reporting evaluation methods (based on Customer Interview Model) as 
currently conducted and as to be modified in the near term (see Recommendations) 
are adequate to support decision making for Tasking/Retasking. 

(a) Evaluation for purposes of Tasking/Retasking need be, and is, 
evaluation of the “Quality” of reporting. 
(b) The attributes of the reporting required to evaluate its “Quality” 
are expressed or implied in the: 
(1) HUMINT Tasking Plan(s) 
(ii) NIT(s) 
(iii) DCID 1/2 
(iv) PRC (1) Directive(s) 
(v) NITC Collection Studies 
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2. The HTO(AB) assessment efforts conducted during this study were based on a 
strategy of providing an “interface” to existing evaluation/assessment efforts 
within the IC. This semi-centralized, interface approach is capable (by 
supplementing existing assessment efforts or providing such effort when it 
does not exist) of supporting Tasking/Retasking (HTO/TB). 


3. The assessment methodology (based on DDO model) as used during this study 
(two specific topic areas) is based on acceptable evaluation techniques: 
(a) objective listing of reporting used by interviewee. 
(b) interview process obtains: 
(1) subjective rating of reporting quality (8 attributes). 
(11) statements of significant short-falls, gaps, and/or overlaps. 
(141i) citations of instances of tangible impacts (policy positions, 
redirected efforts or resources, savings due to redirections, 
"tip-offs", etc.). 


4. Modifications and improvements in the assessment methodology (Customer 
Interview Model) are possible (see Recommendations). Several suggested 
improvements have been tested during this study. 


C. Non-Intelligence Community 


1. Non-IC (other agencies/departments) and non-governmental. evaluation efforts 
have been reviewed and several relevant concepts and methodologies have 
been identified ["market” concept, evaluability assessment (EA) methods, 
survey concepts, etc.). 


2. The state-of-the-art in evaluation relevant to the subject area of this 
study (intelligence reporting and utilization) is “primitive” and evolving. 

(a) Further search for evaluation concepts, tools, and techniques 
should continue, but are unlikely to lead to significant advances. 
Most returns would be repetitions of previously seen concepts, 

7 tools, and techniques. 

(b) Useful concepts, tools, and techniques do exist which can be 
adopted/modified for application to IC evaluation (refer to 
Recommendations following). 

(c) The IC should adopt a strategy of modification and adaptation of 
existing concepts/techniques and minimal innovation. 

(d) Elaborate, complex, and highly sophisticated evaluation tools/ 
techniques do exist, but are not warranted at this stage of 
evaluation of HUMINT reporting in the IC. 


D. Federal Evaluation Policy 


Federal policy is that evaluation of programs be done and included in 
budgetary presentations [OMB Circulars A-1l, A-115, A-117; GAO-“Comprehensive 
Approach for Planning & Conducting A Program Results Review (Exposure Draft)” 
6/78, “Guidelines for Economy & Efficiency Audits of Federally Assisted 
Programs (Exposure Draft)" 6/79]. In addition, such evaluation data is 
expected for use by Congressional review/oversight groups. 


No specific Federal policy/guidance regarding evaluation exists - the 
individual agency/department develops its own techniques within the above 
“general” guidance. Refer to Appendix B for extracts and copies of 
relevant materials. 


ge 
f 
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Recommendations 


HUMINT Assessment System (HAS) 


A HUMINT Intelligence Community-wide (IC) reporting/response assessment 
system (HAS) is recommended. The ad-hoc, supplementary assessment as 
utilized during this special study does not appear capable of supporting 
the HUMINT Tasking System (HTO) when fully operational. 


The HAS should be developed on an interface, semi-centralized basis 
providing the linkage between existing agency, department evaluation 
activities and the assessment required to support HTO. The semi~-centralized 
concept (which has proven successful during this study) would utilize 
existing assessment efforts, and would supplement or provide such effort 
where required. 


The first step in establishment of the recommended HAS should be a formal 

Evaluability Assessment (EA) of the IC HUMINT Tasking and Reporting System 
(partial steps of an EA have been done as a part of this Study) - refer 

to Appendix C for detail on the EA methodology. The recommended EA would: 


(a) refine objectives for the HAS-(What should be expected from it? 
How will its success/failure be established? What will HTO(AB) 
be responsible for?). 

(b) establish whether or not a HAS is plausible (workable) in addressing 
the existing Nationial Tasking and Response/HUMINT System — 
see Appendix D. 

(c) determine whether the intended uses of HAS assessment reporting 
are plausible: 

(i) can/can not support tasking, retasking - HTO/TB. 

(ii) identify and assess other expected uses for HAS 
reporting - budgetary inputs, program management, 
policy, etc. 


The EA and following implementation of a HAS for HUMINT HTO (AB) should be 
viewed as a “test-bed” for evaluation concepts and assessment in the entire 
National Tasking System. 
(a) The concepts being developed and implemented are general and pro- 
bably transferable to other source reporting assessment systems. 
(b) The EA effort is required to address currently voiced concerns 
of assessment activities impact on analyst/user/consumer time. 
This question can not be addressed until a HAS is first defined 
by an EA. 


Assessment Methodology - The Customer Interview Model 


Review survey methodologies related to current HUMINT Assessment survey 
techniques (Customer Interview Model) with a view to limiting the impact 
on analyst/user/consumer time: 


(a) Random sampling and/or rotation of user/consumer community for 
assessment surveys. 

(b) Classification of user/consumer community by “customer type”. 

(c) Retrospective interviews - refering to previous (6-12 months prior) 
responses of interviewee. 
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$ (d) Interviewing via telephone and/or data terminal as a supplement or 
alternative to personal contact. 


2. Standardize aspects of the interviewer questions among the sampled inter- 
viewees so that statistically supportable claims can be made for the sample 
responses (tools and techniques exist which appear directly applicable in 
this regard). 


C. HUMINT [HTO-(AB)] Evaluation - General: 


1. Develop a means to provide an “audit trail" from general guidance (NITs, 
DCID 1/2) to Tasking Plans to specific/field-level reporting requirements to 
reporting to analysis/production to user/consumer and policy/program support. 


2. Define and refine specific attributes of “quality” to be used in evaluation 
of HUMINT reporting responding to Tasking Plans. 

(a) Use the interview process to obtain subjective (user/consumer ) 
quantification of the reporting's performance vis-a-vis the 
attributes. 

(b) Develop a data base of these “quality” measures for analysis and 
decision making (Retasking) use. 

(c) Analysis of this data should be performed to seek differences in 
ratings due to: 

(i) “customer” type 
(ii) time 
(iii) topic, subject 


3. Formalize the surveying of “leads” to tangible impacts and entry of such 
data (on leads and/or impacts) to a data base for possible further use in 
the evaluation of “effectiveness”. 
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c= APPENDIX A - Survey of Assessment Activities (Questionnaire) 


You have been identified as an individual. or group involved in 


assessment activities related to intelligence reporting. The Director 


of Central Intelligence (DCI) has approved a special study of such 
activities as they may relate to human source reporting (HUMINT). 


Please provide the following information about your assessment 
activities: 


l. 


8. 


Name of organizational unit - 
Location/address and contact person(s), with phone numbers - 
Number of persons/staff involved - 
Nature of assessment activities, reporting (please describe) —- 
Where do assessment reports go — 

Primary user(s) - 

Secondry user(s) - 
Diagram/sketch flow of assessment reporting (source to user(s)). 
Do you have any written documentation, guides, procedures, 
references, directives, or other materials related to your 


assessment efforts? 


If yes, please list title and reference data, and indicate 
if copies available. 


Are “representative” copies (samples) of assessment reports 
available? 


Please provide leads to other pertinent groups/individuals 
or activities. , 


Do not: hesitate to include previous, no-longer-active 
assessment activities and their products. 


Please key your answers on separate sheet to above numbers. 


Thank you for your assistance. 
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APPENDIX B ~ Federal Policy Related to 
Evaluation (Extracts) 
The following materials: 


(1) 


Analysis and Justification of Programs 


(Section 24.1 = 24.6) ~- OMB A-ll. 
(2) Zero-Base Budgeting — OMB - 115. 


(3) Management Improvement and the Use 
of Evaluation - A-117. 


provide references to guidance and requirements 


for evaluation that relate to this study report. 
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PREPARATION AND SUBMISSION 
OF BUDGET ESTIMATES (1978) 


Sec. 24.1 


ee 


24.1. Justification of programs and financing. 

All estimates reflected in agency budget submis- 
sions will be supported by szero-base budget 
justifications. 

(a) Material required.—The justification materials 
will consist of decision unit overviews, decision pack- 
ages, and a ranking sheet (see exhibits 24A, 24B, and 
24C). A decision unit overview and set of decision 
packages will be prepared and submitted to OMB 
for each decision unit identified by the agency in 
consultation with OMB (see section 11.4). 

(1) Decision unit overview.—The overview pro- 
vides information necessary to evaluate and 
make budget decisions on each of the decision 
packages in a set without repeating the same 
information in each package. If appropriate, 
summary data may be presented in the 
overview. ; ; 

The decision unit overview will be limited 
to three pages and prepared in the format of 
Exhibit 24A. The overview will provide the 
following information: 

(A) Heading.—Include sufficient informa- 

tion to identify the decision unit. 
Agencies will assign a unique four- 
digit numerical code to each decision 
unit. At a minimum, specify the de- 
partment or agency title, bureau or 
other organizational subunit, title of 
the appropriation or fund account that 
finances the decision unit, the ac- 
count identification code (see section 
21.4), the title of the decision unit and 
the four-digit decision unit code. 

(B) Long-range goal.—Identify the long- 
range goal of the decision unit. Goals 
will be directed toward general needs 
and will.serve as the basis for deter- 
mining the major objective(s) . 

(C) Major objective(s).—Describe con- 


cisely the major objectives of the deci- | 


sion unit, including those set forth in 
the basic authorizing legislation, and 
the requirements these objectives are 
intended to satisfy. 

(D) Current method of accomplishing ob- 
jective(s).—Briefly describe the meth- 


od currently used to accomplish the — 


major objectives of the decision unit. 
(E) Alternative(s).—Briefly describe fea- 
sible alternatives to the current meth- 
od of accomplishing major objectives. 
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(H) 


Analysis and Justification of Programs 


Compare the alternatives in terms of 
anticipated effectiveness and efficiency 
in accomplishing the major objectives 
and cite the sources from which the 
evaluative information is taken. In- 
clude, where appropriate, a brief dis- 
cussion of desirable or undesirable side 
effects, organizational structures and 
delivery systems, longer-range costs, or 
other factors relevant to the selection. 
Specify which, if any, alternative rep- 
resents the method proposed for the 
budget year. When any enlarged or 
new activity is proposed, state why the 
need cannot be filled by State or local 
governments or the private sector. 
Accomplishments.—To the extent pos- 
sible, using specific measures of ac- 
complishment, workload, effectiveness, 
and efficiency, describe the recent 
progress of the decision unit in achiev- 
ing its objectives. The key indicators 
by which outputs and achievements 
will be measured should be obtainable 
from existing evaluation systems, in- 
cluding. periodic in-depth program 
evaluation and work measurement 
systems (see subsection 24.1(b) below). 
Indirect or proxy measures should be 
used only while evaluation and work 
measurement systems are being de- 
veloped. Progress toward short-term 
objectives may also be indicated. In ad- 
dition, a brief narrative may be in- 
cluded to describe accomplishments 
that are not indicated by any of the 
measures. 

Resource summary (optional) .—Sum- 
marize the financial and personnel re- 
sources required in the budget year to 
support each package within the deci- 
sion package set. Compare these 
amounts to the resources required for 
the past and current year. 
Performance, activity, and/or work- 
load summary (optional) —Summarize 
the marginal impact that each decision 
package will have in the budget year 
on key performance, activity, or work- 
load measurements. Compare their im- 
pact to the levels for the past and cur- 
rent years. 
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(2) Decision packages.—Decision package sets will 
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contain packages representing: (1) a mini- 
mum level; (2) the current level (unless the 
total requested for the decision unit is below 
the current level); and (3) an enhancement 


' level, if there is a clear need. Packages rep- 


resenting intermediate levels should be pre- 
pared when significantly different perform- 
ance levels between the minimum and cur- 


_ rent level can be clearly identified. In addi- 


tion, packages corresponding to certain speci- 
fied levels will be included in the set when re- 
quired by OMB representatives. Only one de- 
cision package need be prepared in situations 
where there clearly is no discretion in the 
amount of the funds to be spent or in the 
method or level that the activity is to be con- 
ducted (ie. uncontrollable or mandatory 
activity). 

Enactment of proposed supplementals and 
rescission proposals will be assumed. Amounts 
proposed for supplementals and rescissions 
will be separately identified for the current 
year and merged into the amounts for the 
budget year in the respective decision pack- 
ages. However, for purposes of justification 
and review, separate decision packages will be 
required for each proposed program supple- 
mental. New programs or activities (e.g., those 
resulting from proposed legislation or a new 
major objective) will form the basis for & 
separate decision unit and will be presented 
in a separate decision package set. Proposals 
for termination of current programs or activi- 
ties will also be reflected in a separate deci- 
sion package and will be discussed in the 
summary and highlight memorandum (see 
section 22.1). ; 

The minimum level decision packages 
should not exceed three pages. The other 
packages (intermediate, current, and en- 
hancement) should be limited to two pages. 
However, supplementary material deemed 
necessary to support the decision packages 
may accompany the decision packages. Each 
package within a decision package set will 
clearly describe the increase in activity, fund- 
ing, and performance that would result from 


" {ts inclusion in the budget. Packages will be 


prepared in the format of exhibit 24B and 

provide the follo information: — 

eas SANS re the organizational 
unit (e.g., agency and bureau title), ap- 


propriation or fund account title and 
identification code, decision unit title, 
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the agency four-digit decision unit 
code, and the package number (e.g., 
lof 3). 

Activity —Describe the additional work 
to be performed or services tc be pro- 


‘ vided with the resources specified in 


the package. For packages below the 
current level, the activity description 


’ should provide an explanation of what 


(Cc) 


D 


~ 


(E) 


will be performed, rather that what 
will be eliminated from the current 
level. : 

Resource requirements—Include ap- 
propriate information, such as obliga- 
tions, offsetting collections, budget au- 
thority and outlays, and employment 
(full-time permanent and total) for 
the three years covered by the budget. 
For each measure used, the budget year 
increase associated with the package 
should be listed along with the cumula- 
tive amount for this and all preceding 
packages for the decision unit. Budget 
authority and outlay estimates will also 
be provided for 4 years beyond the 
budget year consistent with the cri- 
teria set forth in section 23. 

Short-term objective(s).—State the 
short-term objectives (usually achiev- 
able within one year) that will be ac- 
complished and the benefits that will 
result from the cumulative resources 
described in the package. The expected 
results of work performed or services 
provided should be identified to the 


“maximum extent possible through the 


use of quantitative measures. All pack- 
ages in a decision package set should 
use the same set: of short-term objec- 
tives and measures, so that the incre- 
mental benefits from one package to 
the next can be readily determined. For 
packages below. the current level, the 
short-term objective should provide a 
description of what will be accomp- 
lished, rather than what will be elimi- 
nated from the current level. 

Impact on major obdjective(s) —De- 
scribe the impact on the major objec- 
tive(s) or goals of both the additional 
and the cumulative resources shown in 
the package using, to the extent pos- 
sible, the same quantitative and quali- 
tative measures for all packages in the 
set. 
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(F) Basis (for minimum packages only) — 
Describe the basis for the minimum 
level, explaining why this level was 
determined to be the minimum level. 

(G) Other information.—Include other in- 
formation that aids in evaluating the 
decision package. This should include: 

(i) Explanations of any legislation 
needed in connection with the 

_ package; 

GD The results of benefit/cost and 
other analyses and evaluations 
that contribute to the justifi- 
cation; 

(iii) For the minimum level package, 
the effect of zero-funding for 
any program in the decision 
units 

(iv) For packages below the current 
level, an explanation of what 
now is being accomplished that 
will not be accomplished at the 
lower level; and 

(vy) Any special relationship of this 
decision package to other deci- 
sion packages, including special 
coordination that may be re- 
quired. If adoption of a decision 
package will affect the output 
of another decision unit, de- 
scribe the expected impact in 
terms of the measures used for 
the affected unit. 

(3) Ranking sheet.—Each agency will prepare and 
submit a ranking sheet in the format of ex- 
hibit 24C that indicates in a single list the 
priority of all decision packages comprising the 
budget request. Additional decision packages 
prepared to examine alternative methods or for 
other purposes that are not part of the agency's 
budget request will not be included in the 
ranking. The ranking sheet will contain the 
following information: 

(A) priority rank number; 
(B) decision unit title and package number; 
(C) budget outlay and total employment 
amounts for the decision package; and 
(D) cumulative budget outlay and total em- 
ployment amounts for the decision 
package and all higher ranked packages. 
In instances (e.g., revolving funds) where 
outlays and total employment are not factors 
in determining the appropriate or priority 
level of performance, agencies should indicate 
@ measure such as budget authority, total obli- 
gations or costs. 
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Sec. 24.1 


(0) Derivation of amounts requested.—Agencies 
should base justifications on or be prepared to sub- 
mit additional information covering the following: - 

(1) Detailed analyses of workload, unit costs, pro- 
ductivity trends, the impact of capital invest- 
ment proposals on productivity, changes in © 
quality of output, and demonstrated results 
of past program activities. . 

(2) Basis for distribution of funds (i.e., formulas | 
or principles for allocation, matching, the 
‘awarding of loans, grants, or contracts, etc.) 
and data on resulting geographic distribution ~ 
(e.g., by State, etc.) with identification of any | 
issues. 

Work measurement, unit costs, and productivity 
indexes should be used to the maximum extent prac- 
ticable in justifying staffing requirements. 

Properly developed work measurement procedures — 
should be used to produce estimates of the staff-_ 
hours per unit of workload, such as the staff-hours 
per claim adjudicated, staff-hours per staff main- 
tained in the field, staff-hours per infested acre of 
pest control, etc., depending on the nature of the 
agency programs. These estimates should represent 
an acceptable level of performance based on current 
realistic time standards. If the agency does not have 
a work measurement system that provides this type 
of information, statistical techniques based on his- 
torical employment input and work output may be 
used, while an appropriate system is under develop- 
ment. 

Unit costs relate the volume of work to the funds 
required to produce the work. Unit costs may in- 
clude, in addition to personnel costs, the costs of 
supplies, travel, equipment, etc. Thus, unit costs re- 
flect the ratio of personnel, materials, travel and 
other costs to the output produced, and will be 
stated in the dollars (or cents) required to produce 
a unit of work. When unit costs include personnel 
costs, work measurement should be used to support 
the acceptability of this component. 

Productivity indexes are based on the ratio of 
total output to resource input. Output measures are 
based on the volume of products or services produced 
for use outside the organization, with due allowance 
for differences in the nature of individual products 
or services. Measures of input may be based on the 
amount of personnel alone, on personnel costs, or 
on & more comprehensive measure of resource inputs 
that includes nonlabor costs. 

Agencies will extend the use of work measure- 
ment and unit cost analysis to both common service 
activities and program activities. Usually, produc- 
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tivity indexes are based on organization-wide totals i 
of both outputs and inputs, thus already covering ! 
both direct and indirect costs. OMB will, to the ! 
extent possible, assist agencies in the establishment 
or improvement of work measurement and produc- 
tivity analysis systems. t 
24.2. Analysis of economic impact. i 

Agencies are reminded of their obligation to pre- 
pare analyses of economic impact whenever they are 
required by criteria developed in accordance with 
OMB Circular No. A-107, and Executive Order 
No. 11949. In addition, agencies should respond to re- 
quests from the Council on Wage and Price Sta- 
bility pursuant to their responsibilities for reviewing 
the economic effects of programs and activities. 
Also, whenever agencies are required by statute to 
prepare economic impact analyses, agencies should 
be prepared to provide these analyses when re- 
quested by OMB representatives. 


24.3. Analysis of paperwork burden. 

Paperwork generated in connection with on- 
going programs will be reviewed and, whenever pos- 
sible, reduced. For every new program initiative or 
expansion of an existing program, an analysis of 
the paperwork burden created for the private sec- 
tor, State and local governments, as well as within 


the Federal establishment, will be submitted with the 


justification. 
24.4, Explanations relating to supplemental estimates. 
‘When the need for a program supplemental appro- 


priation is forecast (see sections 11.6 and 13.2), @ 
decision package set should be prepared in accord- — 
ance with section 24.1. The “Other information” sec- ° 


tion on the decision package should set forth the 
reasons for the omission of the request from the reg- 


war estimates for the period concerned, and the rea- - 
sons why it is considered essential that the additional . 


appropriation be granted during the year instead of 


obtaining the money in a regular appropriation the - 


following year..Whenever possible, requests for sup- 


plementals, whether under existing or proposed legis- 


lation, should be accompanied by recommendations 
as to where corresponding offsets can be made else- 


where in the agency. If the estimate is approved for -- 


later transmittal (rather than in the budget), further 
justification of the supplemental estimate will be 
required when it is formally submitted (see section 
39). In every case, the effect of requested supple- 


mentals will also be shown in the appropriate deci- | 


sion packages (see section 24.1(a) (2)). 

For anticipated supplementals in the current year 
to meet the cost of pay increases, decision packages | 
need not be prepared. However, information should ! 
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be provided identifying, for each appropriation or 
fund, the total cost of the pay increases and the 
amount that is being absorbed in accordance with 
related policy guidance. Any difference from in- 
formation submitted with the apportionment request 
for the current year should also be explained. 


245. Rental payments to GSA. 

Each agency making rental payments in excess 
of $1 million to GSA for space (leased and Govern- 
ment-owned) and buildings services will provide a 
summary report in the form of exhibit 24D. The pay- 
ments reported will include both the'Standard Level 
User Charges (SLUC) and payments for reimburs- 
able special services—e.g., extra guarding or clean- 
ing of buildings—that agencies have requested in 
addition to the normal services provided. 

While the report submitted to OMB will be for the 
agency as a whole, space rental charges should be 
distributed for budget presentation purposes among 
the appropriations and funds that finance the ac- 
tivities utilizing the space. These charges must be 
consistent with amounts reported under object 
classification 23.1 for the affected accounts. Agencies 
should be prepared to provide this information at the 
appropriation and fund account level, if requested. 

Explanations for the information requested are as 
follows: 

(a) SLUC and reimbursable payments —Separate 
estimates should be provided for: 

(1) Standard level user charges (SLUC) to GSA 

for space and related services; and 

(2) Reimbursable payment for special services 

beyond the standard level of services—e.g., 
_ extra guarding or cleaning. 

For the current and budget years, SLUC estimates 
should separately identify base costs and erpansion 
costs. Current year expansion costs represent SLUC 
payments associated with expansion space to be ac- 
quired during the current year. Budget year base 
costs represent SLUC payments for current year 
space inventory, plus current year expansion space, 
less planned space reductions. Budget year expansion 
costs represent SLUC payments associated with 
expansion space to be acquired during the budget 
year. 

(b) Personnel estimates.—Estimates will also be 
provided for ‘full-time permanent and total employ- 
ment for the 3 years covered by the budget. In cases 
where current and/or budget year personnel expan- 
sion is not likely to require additional GSA furnished 
office space—e.g., an increase in air traffic controllers, 
park rangers, or overseas employees—a special nota- 
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tion should be made, with past, current, and budget 
year employment estimates separately identified for 
that activity. Also, if there have been significant cut- 
backs in areas not requiring GSA space that would 
distort the relationship between net personnel ex- 
pansion and new GSA space required, a separate 
notation should be included. DOD estimates should 
break out military and nonmilitary functions. 


(c) Space estinates.—Estimates will also be pro- 
vided for the square foot inventory for which SLUC 
payments are made to GBA. These estimates should 
break out the base inventory and expansion space for 
the current and budget years in the same manner 
that such a break out is provided for SLUC payments. 

If subsequent changes made in budget allowances 
result in revised space or personnel requirements, 
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Public Buildings Service, GSA will be notified in 
order to compute a corresponding revision in rental 
payments. An updated report will be prepared and 
sent to OMB immediately upon receipt of such 
changes in budget allowances. 


24.6. Analysis of ADP and telecommunications re- 
sources. 

The materials required by section 24.1 should 
include justification for ADP and/or telecommunica- 
tions requirements in those decision package sets 
where either the estimates for ADP or telecommuni- 
cations to support the decision unit exceed $1 million 
in the budget year, or where the required ADP or 
telecommunications resources are more than 10 per 
cent of the total estimate for the decision unit (see 
section 43). 
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Analysis and Justification of Programs 


24.1. Justification of programs and financing. 

All estimates reflected in agency budget submis- 
sions will be supported by sero-base budget 
justifications. 

(a) Material required.—The justification materials 
will consist of decision unit overviews, decision pack- 
ages, and a ranking sheet (see exhibits 24A, 24B. and 
24C). A decision unit overview and set of decision 
packages will be prepared and submitted to OMB 
for each decision unit identified by the agency in 
consultation with OMB (see section 11.4). 

(1) Decision unit overview.—The overview pro- 
yides information necessary to evaluate and 
make budget decisions on each of the decision 
packages in a set without repeating the same 
information in each package. If appropriate, 
summary data may be presented in the 
overview. 

The decision unit overview will be limited 
to three pages and prepared in the format of 
Exhibit 24A. The overview will provide the 
following information: 

(A) Heading.—Include sufficient informa- 

tion to identify the decision unit. 
Agencies will assign a unique four- 
digit numerical code to each decision 
unit. At a minimum, specify the de- 
partment or agency title, bureau or 
other organizational subunit, title of 
the appropriation or fund account that 
finances the decision unit, the ac- 
count identification code (see section 
21.4), the title of the decision unit and 
the four-digit decision unit code. 

(B) Long-range goal.—Identify the long- 
range goal of the decision unit. Goals 
will be directed toward general needs 
and will serve as the basis for deter- 
mining the major objective(s). 

(C) Major objective(s).—Describe con- 
cisely the major objectives of the deci- 
sion unit, including those set forth in 
the basic authorizing legislation, and 
the requirements these objectives are 

’ intended to satisfy. 

(D) Current method of accomplishing ob- 
jective(s).—Briefly describe the meth- 
od currently used to accomplish the 
major objectives of the decision unit. 

(E) Alternative(s).—Briefly describe fea- 
sible alternatives to the current meth- 
od of accomplishing major objectives. 
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Compare the alternatives in terms of 
anticipated effectiveness and efficiency 
in accomplishing the major objectives 
and cite the sources from which the 
evaluative information is taken. In- 
clude, where appropriate, a brief dis- 
cussion of desirable or undesirable side 
effects, organizational structures and 
delivery systems, longer-range costs, or 
other factors relevant to the selection. 
Specify which, if any, alternative rep- 
resents the method proposed for the 
budget year. When any enlarged or 
new activity is proposed, state why the 
need cannot be filled by State or local 
governments or the private sector. 
Accomplishments.—To the extent pos- 
sible, using specific measures of ac- 
complishment, workload, effectiveness, 
and efficiency, describe the recent 
progress of the decision unit in achiev- 
ing its objectives. The key indicators 
by which outputs and achievements 
will be measured should be obtainable 
from existing evaluation systems, in- 
cluding periodic in-depth program 
evaluation and work measurement 
systems (see subsection 24.1(b) below). 
Indirect or proxy measures should be 
used only while evaluation and work 
measurement systems are being de- 
veloped. Progress toward short-term 
objectives may also be indicated. In ad- 
dition, a brief narrative may be in- 
cluded to describe accomplishments 
that are not indicated by any of the 
measures. 

Resource summary (optional). —Sum- 
marize the financial and personnel re- 
sources required in the budget year to 
support each package within the deci- 
sion package set. Compare these 
amounts to the resources required for 
the past and current year. 
Performance, activity, and/or work- 
load summary (optional) —Summarize 
the marginal impact that each decision 
package will have in the budget year 
on key performance, activity, or work~ 
load measurements. Compare their im- 
pact to the levels for the past and cur- 
rent years. 
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() Basis (for minimum packages only) — 
Describe the basis for the minimum 
level, explaining why this level was 
determined to be the minimum level. 

(G) Other information.—Include other in- 
formation that aids in evaluating the 
decision package. This should include: 

qd) Explanations of any legislation 
needed in connection with the 
_ Package; 

(i) The results of benefit/cost and 
other analyses and evaluations 
that contribute to the justifi- 
cation; 

(ii) For the minimum level package, 
the effect of zero-funding for 
any program in the decision 
unite 

(iv) For packages below the current 
level, an explanation of what 
now is being accomplished that 
will not be accomplished at the 
lower level; and 

(vy) Any special relationship of this 
decision package to other deci- 
sion packages, including special 
coordination that may be re- 
quired. If adoption of a decision 
package will affect the output 
of another decision unit, de- 
scribe the expected impact in 
terms of the measures used for 
the affected unit. 

(3) Ranking sheet.—Each agency will prepare and 
submit a ranking sheet in the format of ex- 
hibit 24C that indicates in a single Ust the 
priority of all decision packages comprising the 
budget request. Additional decision packages 
prepared to examine alternative methods or for 
other purposes that are not part of the agency's 
budget request will not be included in the 
ranking. The ranking sheet will contain the 
following information: 

' (A) priority rank number; 
(B) decision unit title and package number; 
(C) budget outlay and total employment 
amounts for the decision package; and 
(D) cumulative budget outlay and total em- 
ployment amounts for the decision 
package and all higher ranked packages. 
In instances (e.g., revolving funds) where 
outlays and total employment are not factors 
in determining the appropriate or priority 


level of performance, agencies should indicate’ 


@ measure such as budget authority, total obli- 
gations or costs. 


~B4- 


(b) Derteation of amounts requested.—Agencies 
should base justifications on or be prepared to sub- 


mit additional information covering the following: © 


(1) Detailed analyses of workload, unit costs, pro- 
. ductivity trends, the impact of capital invest- 
ment proposals on productivity, changes in 
quality of output, and demonstrated results 

‘of past program activities. 
(2) 


Basis for distribution of funds (.e., formulas | 


or principles for allocation, matching, the 


awarding of loans, grants, or contracts, etc.) 


and data on resulting geographic distribution ~ 
(e.g., by State, etc.) with identification of any 


issues. 
Work measurement, unit costs, and productivity 
indexes should be used to the maximum extent prac- 
ticable in justifying staffing requirements. 


Properly developed work measurement procedures _ 


should be used to produce estimates of the staff- 
hours per unit of workload, such as the staff-hours 
per claim adjudicated, staff-hours per staff main- 
tained in the field, staff-hours per infested acre of 
pest control, etc., depending on the nature of the 
agency programs. These estimates should represent 
an acceptable level of performance based on current 
realistic time standards. If the agency does not have 
a work measurement system that provides this type 
of information, statistical techniques based on his- 
torical employment ipput and work output may be 
used, while an appropriate system is under develop- 
ment. 

Unit costs relate the volume of work to the funds 
required to produce the work. Unit costs may in- 
clude, in addition to personne! costs, the costs of 
supplies, travel, equipment, etc. Thus, unit costs re- 
fiect the ratio of personnel, materials, travel and 
other costs to the output produced, and will be 
stated in the dollars (or cents) required to produce 
a unit of work. When unit costs include personnel 
costs, work measurement should be used to support 
the acceptability of this component. 

Productivity indexes are based on the ratio of 
total output to resource input. Output measures are 
based on the volume of products or services produced 
for use outside the organization, with due allowance 
for differences in the nature of individual products 
or services. Measures of input may be based on the 
amount of personnel alone, on personnel costs, or 
on a more comprehensive measure of resource inputs 
that includes nonlabor costs. 

Agencies will extend the use of work measure- 
ment and unit cost analysis to both common service 
activities and program activities. Usually, produc- 
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tivity indexes are based on organization-wide totals i 
of both outputs and inputs, thus already covering ! 
both direct and indirect costs. OMB will, to the! 
extent possible, assist agencies in the establishment 
or improvement of work measurement and produc- 
tivity analysis systems. i 


242. Analysis of economic impact. i 


Agencies are reminded of their obligation to pre 
pare analyses of economic impact whenever they are 
required by criteria developed in accordance with 
OMB Circular No. A-107, and. Executive Order 
No. 11949. In addition, agencies should respond to re- 
quests from the Council on Wage and Price Sta- 
bility pursuant to their responsibilities for reviewing 
the economic effects of programs and activities. 
Also, whenever agencies are required by statute to 
prepare economic impact analyses, agencies should 
be prepared to provide these analyses when re- 
quested by OMB representatives. 


4.3. Analysis of paperwork burden. 

_ Paperwork generated in connection with on- 
going programs will be reviewed and, whenever pos- 
sible, reduced. For every new program initiative or 
expansion of an existing program, an analysis of 
the paperwork burden created for the private sec- 
tor, State and local governments, as well as within 


the Federal establishment, will be submitted with the : 


justification. 
24.4, Explanations relating to supplemental estimates. 


‘When the need for a program supplemental appro- : 
priation is forecast (see sections 11.6 and 13.2), 8 : 


decision package set should be prepared in accord- 


ance with section 24.1. The “Other information” sec- ‘ 


tion on the decision package should set forth the 
reasons for the omission of the request from the reg- 


ular estimates for the period concerned, and the rea- - 
sons why it is considered essential that the additional : 


appropriation be granted during the year instead of 


obtaining the money in a regular appropriation the : 


following year. Whenever possible, requests for sup- 


plementals, whether under existing or proposed legis- . 


lation, should be accompanied by recommendations 


_ as to where corresponding offsets can be made else- 
. where in the agency. If the estimate is approved for - 


later transmittal (rather than in the budget), further 
justification of the supplemental estimate will be 
required when {t is formally submitted (see section 
39). In every case, the effect of requested supple- 


mentals will also be shown in the appropriate deci- © 


sion packages (see section 24.1(a) (2)). 

For anticipated supplementals in the current year 
to meet the cost of pay increases, decision packages . 
need not be prepared. However, information should ! 
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be provided identifying, for each appropriation or 
fund, the total cost of the pay increases and the 
amount that is being absorbed in accordance with 
related policy guidance. Any difference from in- 
formation submitted with the apportionment request 
for the-current year should also be explained. 


245. Rental payments to GSA. 

Each agency making rental payments in excess 
of $1 million to GSA for space (leased and Govern- 
ment-owned) and buildings services will provide a 
summary report in the form of exhibit 24D. The pay- 
ments reported will include both the'Standard Level 
User Charges (SLUC) and payments for reimburs- 
able special services—e.g., extra guarding or clean- 


! ing of buildings—that agencies have requested in 


addition to the normal services provided. 

While the report submitted to OMB will be for the 
agency as a whole, space rental charges should be 
distributed for budget presentation purposes among 
the appropriations and funds that finance the ac- 
tivities utilizing the space. These charges must be 
consistent with amounts reported under object 
classification 23.1 for the affected accounts. Agencies 
should be prepared to provide this information at the 
appropriation and fund account level, if requested. 

Explanations for the information requested are as 
follows: 

(a) SLUC and retmdursable payments.—Separate 
estimates should be provided for: 

(1) Standard level user charges (SLUC) to GSA 

for space and related services; and 

(2) Reimbursable payment for special services 

beyond the standard level of services—e.g., 
extra guarding or cleaning. 

For the current and budget years, SLUC estimates 
should separately identify base costs and ezpansion 
costs. Current year expansion costs represent SLUC 
payments associated with expansion space to be ac- 
quired during the current year. Budget year base 
costs represent SLUC payments for current year 
space inventory, plus current year expansion space, 
less planned space reductions. Budget year expansion 
costs represent SLUC payments associated with 
expansion space to be acquired during the budget 
year. 

(b) Personnel estimates.—Estimates will also be 
provided for full-time permanent and total employ- 
ment for the 3 years covered by the budget. In cases 
where current and/or budget year personnel expan- 
sion is not likely to require additional GSA furnished 
office space—e.g., an increase in air traffic controllers, 
park rangers, or overseas employees—a special nota- 
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tion should be made, with past, current, and budget 
year employment estimates separately identified for 
that activity. Also, if there have been significant cut- 
backs in areas not requiring GSA space that would 
distort the relationship between net personnel ex- 
pansion and new GSA space required, a separate 
notation should be included. DOD estimates should 
break out military and nonmilitary functions. 


(ec) Space estimates.—Estimates will also be pro- 
vided for the square foot inventory for which SLUC 
payments are made to GSA. These estimates should 
break out the base inventory and expansion space for 
the current and budget years in the same manner 
that such a break out is provided for SLUC payments. 

If subsequent changes made in budget allowances 
result in revised space or personnel requirements, 
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Public Buildings Service, GSA will be notified in 
order to compute a corresponding revision in rental 
payments. An updated report will be prepared and 
sent to OMB immediately upon receipt of such 
changes in budget allowances. 


24.6. Analysis of ADP and telecommuniestions re- 
_ sources, 

The materials required by section 24.1 should 
include justification for ADP and/or telecommunica- 
tions requirements in those decision package sets 
where either the estimates for ADP or telecommuni- 
cations to support the decision unit exceed $1 million 
in the budget year, or where the required ADP or 
telecommunications resources are more than 10 per 
cent of the total estimate for the decision unit (see 
section 43). 
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a Extracted from OMB Circular No. A-115 


> Zero-Base Budgeting - May 5, 1978 (pages 6-9) 


( -. c. Identification of objectives for each decision unit. Meaningful 

; objectives should be established for each decision unit. Both 
major objectives (the main results ultimately expected to be 
accomplished) and short-term objectives (outputs expected for a 
given level of funding during the budget year) should be identi- 
fied. The manager of the decision unit should include in a state- 
ment of objectives information relating to both the services or 
products to be provided by the decision unit and the contribution 
of that output to achievement of the principal missions toward 
which the unit's efforts are directed. 


Major objectives should: 


-—- be explicit statements of intended output clearly related to the 

mission of the program or organization; 

be developed by participation of managers at all levels; 

be subject to realistic measures of performance; . 

form the basis for evaluating the accomplishments of the program 

or organization; and 

-- normally be of a continuing nature, or take relatively long 
periods to accomplish. 


Top level agency management should be involved in setting objectives for lower- 
level agency members to: 


(1) ensure that appropriate policy guidance is furnished to managers 
(— throughout the agency; 

(2) aid managers who prepare decision packages to define, explain, and 
justify their activities and the required resources at different 
levels of performance; and 

(3) aid higher level managers in understanding and evaluating the 
budget requests. 


Some factors to be considered in identifying objectives are: 


-- Number of objectives. The number of objectives for a decision 
unit may have a direct bearing on the effectiveness of the ZBB 
process. Too many may make it difficult for higher management 
levels to assess the relative importance of each decision package 
and its respective ranking in comparison with other packages. 
Moreover, too many objectives for a decision unit may indicate 
that it was established initially at too aggregated a level. 


-- Measures of performance. As objectives are identified, managers 
should similtaneously determine the key indicators by which per- 
formance is to be measured. Agencies should specify measures of 
accomplishment, workload, effectiveness, and efficiency for each 
decision unit. The key indicators by which outputs and achieve- 
ments will be measured should be obtainable from existing evalua- 
tion systems, including periodic in-depth program evaluations, 
and work measurement systems (see OMB Circular No. A-1l1, regard- 

C ing the justification of programs and financing). Indirect or 
: proxy measures should be used only while evaluation and work 
meausrement systems are being developed. 
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-- Organization plans and goals. Objectives should evolve from 
stated plans and goals for the organization. If such plans and 
goals have not already been communicated, this step should pro- 
vide for better understanding between organizational units on the 
purpose and direction of the respective units. 


d. Identification and evaluation of alternative methods of accom 
plishing objectives. Following the establishment of objectives, 
the decision unit manager will examine alternative methods of 
accomplishing each objective. The study of methods of operation 
that differ from existing practices necessitates a rethinking by 
managers of whether the current way of doing business is still the 
most appropriate way. This often requires a reexamination of the 
program, including a review of existing legislation, organization 
structure, and existing managerial practices. It also requires 
managers to search for innovative ways to achieve program objec- 
tives at lower costs. 


Assessments of alternatives should be based on the relative 
effectiveness and efficiency of each alternative in accomplishing 
major objectives, taking into account desirable and undesirable 
consequences of each alternative. If available information is not 
adequate to assess all reasonable alternatives, evaluation efforts 
should be undertaken so that meaningful comparisons can be made 
for future budget cycles. 


In some instances, feasible alternatives may require additional 
legislation, the need for which may have been identified during 
the course of a major reexamination of the program or activity. 

In other instances, it may not be possible to study or to examine 
fully alternative methods within the budget cycle, in which case 
adjustments can be made subsequent to the submission of the 
budget. In still other instances, the alternatives identified may 
represent the first steps toward more significant program changes 
that will take longer than one year to accomplish. 


The alternatives that are identified must be both feasible and 
realistic. They must indicate to reviewers that management is 
making a conscientious and continuing effort to improve the effec- 
tiveness and efficiency of the unit. Fach higher level of review 
should thoroughly examine the alternatives to ensure that 
sufficient attention has been given to ways to improve operations 
or reduce resource requirements. To ensure adequate coverage, it 
may be advisable for central budget, evaluation, and planning 
staffs to participate in the evaluation of alternatives and the 
establishment of implementation schedules for those that are 
selected. 
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ie EXECUTIVE OFFICE OF THE PRESIDENT 


SEES Say 
Kl a of A 
HINER OFFICE OF MANAGEMENT AND BUDGET 
/ WASHINGTON, D.C. 20503 


MAR 23 1979 


CIRCULAR NO. A-117 
TO THE HEADS OF EXECUTIVE DEPARTMENTS AND ESTABLISHMENTS 


SUBJECT: Management Improvement and the Use of Evaluation 
in the Executive Branch 


1. Purpose. This Circular provides guidance on management 
improvement initiatives designed to increase the efficiency 
and effectiveness of program operations in the Executive 
Branch. It also emphasizes the importance of the role of 
evaluation in overall management improvement and the budget 
process. 


2. Rescissions. This Circular supersedes and rescinds 
Circular No. A-44 (Revised), dated May 24, 1972, subject: 
"Management review and improvement program"; and Circular 
No. A-113, dated November 17, 1976, subject: "Preparation 
and submission of management plans." 


3. Policy. All agencies of the Executive Branch of the 
Federal Government will assess the effectiveness of their 
programs and the efficiency with which they are conducted 
and seek improvement on a continuing basis so that Federal 
management will reflect the most progressive practices of 
both public and business management and result in improved 


service to the public. 


4. Definitions. For the purposes of this Circular, the 
following definitions apply: 


a.: Management improvement is any action taken to 
improve the quality and timeliness of program performance, 
increase productivity, control costs, or mitigate adverse 
aspects of agency operations. 


b. Management evaluation is a formal assessment of 
the efficiency of agency operations. It includes assessing 
the effectiveness of organizational structures and relation- 
ships, operating procedures and systems, and work force 
requirements and utilization. 
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c. Program evaluation 1S a formal assessment, 
through objective measurements and systematic analyses, 
of the manner and extent to which Federal programs (or 
their components) achieve their objectives or produce 
other significant effects, used to assist management and 

_ policy decisionmaking. 


d. Performance measures are reliable and objective 
indicators of the results of a Federal activity. They 
include measures of efficiency, effectiveness, program 
impact, and program output or workload. 


e. Productivity is a ratio between the output of an 
organizational unit and the input it utilizes during a 
specified period of time. Output is expressed in terms 
of items produced, services provided, or some other measure 
of achievement. Input is expressed by the most relevant 
measure of resources consumed by the unit, such as 


personnel effort, equipment, or total dollars. 


£. Evaluation resources in a department or agency 
are the funds and personnel used to conduct management 
evaluations and audits, program evaluations, and 
productivity measurement. ) 


5. General guidelines. The heads of all executive 
departments and agencies are responsible for developing 
and pursuing comprehensive management improvement efforts. 
The objective of such efforts should be discernable 
improvement in Federal programs--in the efficiency of 
administration or management and in the effectiveness of 
results. The basis for identifying management improvement 
needs is a sound evaluation system. While agency evaluation 
systems may serve multiple purposes, they should have the 
following characteristics to be effective in contributing 
to management improvement. 


a. They should focus on program operations and 
results. They should include procedures to assure that 
evaluation efforts result in specific management improve- 
ments that can be validated. 


b. They should assist management in the identifica- 
tion of program objectives, in providing explicit 
statements of intended outputs related to the objectives, 
ana in developing realistic performance measures to be 
used in conducting evaluations. 
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c. They should be relevant to the budget process 
in that evaluation results should be a major input to 
resource allocation decisions. 


d. Agency evaluation activities should be reviewed 
periodically to assure that: 


-- evaluation efforts that do not contribute 
directly to improving program operations or 
facilitating their improvement are minimized; 


~- there is a balanced emphasis on both evaluation 
- and prospective analyses, such as planning and 
policy analysis; and 


-- they use available evaluation resources in an 
economic and efficient manner. 


Continuing attention should he paid to management improve- 
ment and cost reduction opportunities in activities such 
as accounting, ADP operations, cash management, communi- 
cations, data collection, grants management, paperwork, 


printing and reproduction, regulations improvement, 


travel, ana other administrative activities. 


6. Reporting requirements. Every department and agency 
whose budget is subject to review by OMB will submit an 
annual report to OMB, in accordance with the attachment 
to this Circular, summarizing the resources devoted to 
management improvement and evaluation activities and 
naming the principal officials responsible for those 
activities. 


7. Responsibilities of the Office of Management and 


Budget. In addition to the actions taken as part of the 
budget process, OMB, as part of its management responsi- 
bilities, will: 


a. Seek to identify areas or operations of the 
Federal Government in which significant management 
improvements can be achieved and the steps necessary to 
accomplish those improvements; 
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b. Assess the effectiveness of agency management 
improvement and evaluation activities anc conduct tcliow- 
up consultations with departments and agencies as 
appropriate; 


c. Disseminate information about highly successful 
Management improvement projects in areas of government- 
wide interest; 


dad. Promote the development and use of valid per- 
formance measures; and 


e. Conduct or sponsor a limited number of management. 
improvement projects auring the year. The projects will 
be assigned by the Director, OMB, and generaiiy will be 
of Presidential interest. 


8. Presidential Management Improvement Awards. Awards 
will be made by the President to a limited number of 


individual Federal employees, teams or organizational 
units who have made exceptional and outstanding contri- 
butions to the improvement of management in the Executive 
Branch. Details of this program are contained in Chapter 
451 of the Federal Personnel Manual. 


9. Implementing Instructions. Copies of any instructions 
implementing this Circular should be forwarded to the 
Director, Office of Management and Budget, Attention: 
Assistant Director for Management improvement and Evaluation. 


10. inquiries. iIncguiries relative to this Circuiar should 
be directed .o the “7ffice of Management and Budget, 
Management liprovement anc Evaluation Division, Room 10235, 
New Executive Office Building, Washington, D.C. 20503, 
telephone (262) 395-4960 or 395-5642. 


. —7 Me j - i 
egeetg. et fl = Prlgre, 7. 
James T. McIntyre, Jr of 
(Birector id 
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ATTACHMENT 
Circular No. A -117 


— 


INSTRUCTIONS FOR ANNUAL REPORT TO OMB 


On or before June 30, 1979, and by May 15 of each subsequent 
year, every department or agency whose budget is subject to 
review by OMB will submit a report of resources for evalua- 
tion and management improvement, using the format of the 
attached exhibit. 


Fiscal Year - Enter the current fiscal year. 


Department or agency - Enter the name of the department or 
independent agency submitting the report. 


Bureau - Cabinet departments are to submit a separate sheet 
for each bureau or comparable organizational unit, plus a 
separate sheet giving department totals. Independent agencies 
submit a single sheet giving agency totals only. 


Resources (Obligations) - For each line enter the total obli- 
gations, in thousands of dollars, expected to be devoted to 
that function during the current fiscal year. Include in the 
totals only obligations for personnel compensation, personnel 
benefits, contracts, and grants. 


Resources (Staff Years) - For each line enter the total esti- 
Mated full-time equivalent staff years devoted by all agency 
employees (both full-time and part-time) to that function 
during the current fiscal year. Include professional, admin- 
istrative, and clerical staff, but do not include any employees 
who will spend less than 25 percent of their time on the func- 
tion during the fiscal year. 


LINE ENTRIES 


For each function listed, enter obligations and staff years 

as indicated. Do not include any resources in more than one 
category. Prorate among categories if necessary. For example, 
if a particular study is 70% program evaluation and 30% man- 
agement evaluation, include 70% of the study resources under 
program evaluation and 30% under management evaluation. 


For each line entry, supply the name, title, address, and 
telephone number of the principal agency official directly 
-responsible for and familiar with the details of the activi- 

ties included. 
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1. Management Evaluation - Enter total obligations and staff 
years devoted to management evaluation as defined in Circu- 
lar A-l17. — 


For the purposes of this report, management evaluation acti- 
vities consist of the following: 


~ (1) Planning, developing, assessing, and modifying 
organizational structures and relationships; 


-- Operating procedures; 


-- Internal regulations, instructions, and 
delegations of authority; and 


-- Management information and control systems 
(but not including normal operation and 
maintenance of such systems). 


(2) Conducting or guiding 


-- Assessments of operating efficiency or 
effectiveness; and 


-- Analyses of specific administrative needs. 


(3) Assessing worker productivity (but not including i 
routine collection and processing of productivity 
data), achievement of performance objectives, and 
other quantitative measures of operational effi- 
ciency and effectiveness. 


2. Program Evaluation - Enter total obligations and staff 

years devoted to program evaluation as defined in Circular A-ll7Z. 
In the spaces provided, give a breakdown of obligations into 
personnel costs (compensation and benefits) and contracts and 
grants. 


For the purposes of this report, the following are program 
evaluation activities: 


(1) Formal studies, surveys, and data analyses for the 
- purpose of determining the effectiveness, efficiency, 
or impact of a national or regional program. 


(2) Systematic assessment of demonstration programs or 
projects which are expected to have major implica- 
tion for programs of national or regional scope; 
except that evaluation activities which are an in- 
trinsic part of the program operations or management 
should not be reported. 
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(3) 


44) 


(5) 


(6) 


Assessment and development of program designs to 
assure that programs, once operational, can be 
successfully evaluated. 


Design, development, and field testing of new 
program evaluation methodologies. 


Synthesis and further analysis of resuits 
obtained by several previous program evaluation 
efforts. 


Collection of initial data to help in evaluation 
design and provide a baseline for subsequent 
evaluations. 


The following are not to be included in program evaluation 
activities for this report: 


(1) 
(2) 


(3) 


(4) 


(5) 


Design, development, and operation of general data 
systems or management information systems. 


Continuing collection of routine data ana general 
purpose statistics. 


Analysis of existing or proposed policies where no 
programs yet exist (even though authorized) for 
purposes of appraising the likely costs and effects 
of feasible alternatives. Althcugh such analyses— 
are oiten called “evaluations,” they are prospective 
in character; whereas program evaluation is retro- 
spective, aimed at determining what has actually 
occurred as the result of past program actions. 


Basic research and studies intended to increase 

or foster general knowledge development, but which 
are not expectecé to be used specifically and pri- 
marily in policy anc management decisions. 


Routine, day-to-day monitoring of program operations 
which is an intrinsic part of program administration. 


Productivity Measurement - Enter total obligations and 


staff years devoted to work measurement, determination 
of unit costs, and the collection and processing of other 


' data whose primary use is to measure the productivity of 


the agency's own operations. Include routine measurement 
activities as well as special studies. 
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Other Management Improvement - Enter total obligations 
and staff years devoted to management improvement, as 
defined in Circular A- 117, which are not included in 


the other line entries. 


Total Department/Agency/Bureau - Enter the total obli- 


gations and staff years devoted to the management and 
evaluation functions detailed in the line entries above. 
The line entries should add to the total shown. 

PREVIOUS YEAR UPDATE 


Beginning*with the May 15, 1980 report, the resources 


actually devoted to evaluation and management improvement 


activities in the department or agency for the previous 


fiscal year should be reported. However, no update 1s 


necessary if the actual department or agency totals are 


within 10% of the previously reported estimates. 


SUBMISSION 


Reports for all departments and agencies covered by this 
requirement shall be submitted by May 15 each year to OMB, 
attention:. Management Improvement and Evaluation Division. 
Inquiries concerning this report should be addressed to 
the Division (395-5193). 
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EXHIBIT : 
Circular A~1]}]7 


FISCAL YEAR * 
RESOURCES FOR EVALUATION AND MANAGEMENT IMPROVEMENT 


Department or Agency: ; a . Date of submission: 


Bureau*: ' Check one: Current year estimate 
Previous year update** 


MANAGEMENT IMPROVEMENT RESOURCES RESPONSIBLE OFFICIAL 
AND EVALUATION Obligations Staff Years Name, Title, Address, 
FUNCTION (Thousands of dollars) (PTE) Telephone Number 


l. Management 
Evaluation 


2. Program Evaluation 
(Personnel) ( ) 
(Contracts & Grants) ( ) 


3. Productivity 
Measurement 


eae ORE A 


4. Other Management 
' Improvement 


TOTAL R&SOURCES: 


i 


* Cabinet Departments submit a separate sheet for each bureau or comparable subunit, plus one sheet giving 


Department totais. Independent agencies submit a single sheet giving agency totals oniy. 


** Actual resources for the previous year should be submitted only if they atster, by more than 10 percent 


wfrom the previously reported estimates for that year. 
_ Approved For Release 2005/01/10 : CIA-RDP83M00171R001500070010-4 
-B17- 


Se I eee 
Approved For Release 2005/01/10 : CIA-RDP83M00171R001500070010-4 
% APPENDIX C - Evaluability Assessment (EA) 
“Evaluability Assessment (EA) is a descriptive and analytic process 
intended to produce a reasoned basis for proceeding with an evaluation that 
is likely to prove useful to both management and policy makers.” [Schmidt, 1978] 


EA considers and documents the following: 


(a) the system objectives, expectations, and causal 
assumptions of policy-makers and managers in 
charge of the program. 


(b) what political groups (Congress, Executive Branch 
policy-makers, and interest groups) say the system 
objectives are. 


(c) the extent to which management's system objectives 
and information needs have been defined in 
measurable terms. 


(d) the system activities actually under way. 


(e) the likelihood that system activities will achieve 
measurable progress toward system objectives. 


(f) likely uses of information on system performance. 


(g) options for changes in system activities, objectives, 
and use of information that could enhance system 
performance. 


The final products of an EA would be: 


(1) a written list of agreed-on system objectives, important 
side-effects, and measures (indicators of performance to 
which the system managers could realistically be held 
accountable. 


(2) a set of evaluation/management options of ways that manage 
ment can change system activities, objectives, or uses of 


information (measures of performance) so as to improve 
system performance. 


A. EA Origins 


The EA methodology grew out of efforts of the program evaluation 
group at the Urban Institute during the last ten (10) years. Many of these 


efforts were Federally funded studies - in particular through DHEW. 
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B. EA Assumptions 


The assumptions related to EA are: 


Research 


(1) 
(2) 


(3) 
(4) 
(5) 


evaluation is 


the environment of governmental systems (programs) is complex. 


objectives of such systems are unclear, cloudy, poorly 
defined, ambigious, and vague. 


results of system activities are often uncertain. 
demand for evaluation is growing. 


problems in response to demands for evaluation: 
- purpose of evaluation unclear. 
- few examples of evaluation resulting in 
more effective performance. 
- usefulness of evaluation products questioned 
by both officials and evaluators. 


and experience in the field of evaluation has shown that 


likely to aid in improving system (program) performance if 


following three criteria are met: 


(1) 


(2) 


(3) 


system (program) objectives are well defined —- 
specific measures of performance and data on those 
measures exist and are obtainable at a reasonable cost. 


system assumptions/objectives are plausible - 
evidence exists that system activities have some 
probability of causing the results/effects stated 
in the system objectives. 


intended uses of evaluation (assessment) information 
are well defined and plausible - 

the system managers have defined the uses of 
evaluation information and such uses are plausible. 


the 


EA is a preliminary assessment of a system (program) to determine if the 


above three criteria are satisfied. The major assumption is that if these 


criteria are not met, then further efforts to evaluate the system or activity 


would be fruitless. 


C. EA Experience/Applications 


EA studies have been conducted in the following Federal or government 


agencies: 
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ie () 
. (2) 
(3) 

(4) 


National Institute of Mental Health (NIMH) - 1974. 


National Institute of Law Enforcement and Criminal Justice - 1977. 


Appalachian Regional Commission - 1976. 


Bureau of Health Planning and Resources Development — 1977. 


The above four (4) cases are discussed in J.S. Wholey, Evaluation: 


Promise and Performance, Urban Institute, 1979 and further references are 


provided. 


D. Applicability of EA to Intelligence Community 


1. A formal Evaluability Assessment (EA) of the HUMINT Tasking and Reporting 


System 


is recommended. There are two (2) levels at which an EA can be 


conducted - both are recommended: 


(a) 


(b) 


Tasking Plan level - the system supporting the development 
and production of Tasking Plans and their later retasking. 
This is the set of activities that make up the HTO(TB) and 
HTO(AB) portion of the HUMINT Assessment System (HAS). 


National HUMINT Tasking System (NHTS) or System level - 

the overall system (which includes (1) above) related to 
Intelligence Community (IC) tasking of various agencies/ 
departments, their tasking of collection resources, collection 
activities, reporting, analysis (consumer), final users 
(policy-makers), assessment/evaluation, and retasking. 

This is the total HUMINT Assessment System (HAS) supporting 
HTO. 


2. The EA would accomplish the following for both system levels above - 


Plan le 
(a) 
(b) 


(c) 


It is 
first as a 


made on the 


vel and System level: 

provide well defined system objectives. 

establish plausibility of system assumptions/objectives 
(system activities have probability of causing results/ 


effects stated in objectives). 


determine plausibility of intended uses of evaluation and 
assessment reporting. 


recommended that the Plan level EA be conducted and completed 
test/experimental case for the System level EA. Progress has been 
Plan level EA. 
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Reporting uses, impacte, shortfalls - TP 
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Amount of redefinition of TP objectives ~ Tha 


amount of rerasking ~ TP objectives sad Te 
Collection Managenent 
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Geogcaphic ares, region, country 
Priocity 
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Organizations’ objectives =~ national 
incelligence objectives 
Organization's objectives — TP objectives 
Operating directives < Orgenisacion 
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Use of finished incelligence in policy andj 
of program support and/or setion 
Use of reporting dicactly in policy aad/or 
program support and/or action 
Redirection or aaving of efforce and/or funds 
thac 1s traceable co che HUMINT Seporting 
Shorefell in contributica co all the above 
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Collector 
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Collector 
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Collector 
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APPENDIX E - HUMINT Response Assessment Objectives 


1. 
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To ensure the subject and geographic coverage by HUMINT 
that is required for that type of intelligence by 
national need, as specified in HUMINT Tasking Plan 
objectives and associated amplifying requirements. 

To ensure adequate HUMINT responsiveness to the priority 
dependence upon it, in meeting national need, that is 
specified in Tasking Plans in connection with certain 
objectives. 

To ensure an amount and quality of HUMINT that constitute 
sufficient response to the amplified objectives in 
Tasking Plans, in accordance with the collection 
responsibilities assigned, as revealed in the uses and 
impacts of the reporting. 

To identify significant shortfall in timely, quality 
HUMINT response to amplified objectives in the Tasking 
Plans, as revealed in the observations of key consumers 
and users of HUMINT. 

To obtain the consumer/user feedback required in the 
above in such a way as to ensure its reliability and 
availability. 

To obtain this feedback through a system that is 
cost-effective and help spread new emphasis upon 
effectiveness measurement with respect to HUMINT 


collection and its assessment. 
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